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ABSTRACT 

Unobtrusive measures are recommended as a means of 
assessing edueational outcomes of colleges # Such measures can 
eounteract the response bias whiah is common in questionnaires and 
intervieifs. Outcomes researchers are, in fact, asked to supplement 
standard measures with unobtrusive measures* Interesting data may 
result from observation of students* social interaction and seatii 
patterns, or the tenor of student graffiti and campus bulletin 
boards* Unobtrusive measures are warranted on the measurement grounds 
of reliability due to multiple measures, or increased sampling* In 
addition, observation of behavior is a more valid measure of 
attitudes than an interview in which the responses may be biasaSp 
Unobtrusive measures may be much less costly than standard tests, 
since conclusions may be drawn from data which have already been 
collected and stored • The prudence of unobtrusive measures is 
demonstrated by the wisdom of using multiple measures to assess 
something as multifaceted as the quality of a college education* Some 
eumples of this type of measurement include the analy^ir of 
students* course-^taki/ig patterns* (GDC) 
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The Cme for 
Unobtrusive Measures 



I hvtv iMH he \\U\v iioubl \hM much of vvhtil we know Uom thi- hoeitd 
t*d<*fH<M* h'lt* bt*uh tk*vi'li?ptHi irom Mi^rvww or t|iitM»H0nnairi* clsii^i. But 
n4ial tan :uiy aboul the fufchiy ui tho^c piHiijiih for the 
4jtMl *'diH:MUjf\'^] bt'liiUiufH and pUvntmwm thry ilvptdl Cimsklvt 

w.rcMMrch in infr I hginicr I railing %how{s) th*it depiMidiibh* f^iims m 
lrshp*Hsin^» ^ibihly {cm} hv itMvd h> vHpvuvmv wllfi (neviuiH 
even wliere no knowledge of ref^ults (hiisl boen pruvidiuh ,,Sifni!ft! 
gnins havi* been shoivn in personal adjusimenl' scores (Webb el aL 

Male interviewers obhiin fewer response?* Hi.in fern*'ihj, ^nd fewe^l of 
iill from nmles, while female ink*r viewers obUiin Uieir highest 
f e§pun?^e^ from iiien, eKtep^ for young women hilking io youn^ men 
(Bcnney, Rlcsniiia k Starr, p. 143). 

Sequences of question*! disked in very similar formal produce ^lereo- 
typed response**, nuch as *i tendency to endorse the righlhnnd or the 
leflhand response, or to iilterntite in some simple fa^hiDn. Furl her- 
more, decreasing* t^lention produces reliiible biases from the order of 
item presentation (Webb et igOd, p.io). 

Thus, much of what we know m^y be biased in various and sometimes 
unknown ways. But If what one blind man learns* about elephants is biased 
by the data-gathering procedures adapted, measurement and 'dimpling 
theory suggest it is reasDnable to expect that (he evidence gathered by 
multiple blind men, when pooled, will give a better, if imperfect, approK- 
imtition of an elephant. There is. after all more than one way of knowing. 
The Central thesis of this paper is that multiple research designs and 
measures of educational outcomes arc more likely to yield reliable and 
valid assessments of educaiional outcomes than is the current reliance on 
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Camtdvr \)w fo!lowiitg: 

The wmr in Hie floor tiles In Chic*igo^s MuHiHHii of $vmu:v md 
Industry, 

Tlie Bhf inking dmrneler of a circle of sealed rhii Jren. 
Pupil dikiion In [hv vyvH of Jiidr cusiomrrs. 
Thv bullfiglilt*r'r, beard. 

Riich of ihvw condllion^ cm, under cerliiin c irciirnBliincr" ^ iAm m 
^imvmuw of a phvmimemn of interest to sortieone. Mivi^ Vdih el 
al iiQ66l mch m example of what has came to be Oilf J ^Hruslve 
me^^ureW' & general class of measuremenl!* presumed to fiduce or ellmi- 
n.iit^ ihv pntfmfial fm him: responses undwadcristic of the 

altitude or behavior outHlde the measurement situatlmi and induml by 
the mensurernent art ii^rlf. The f^remisr h (hat when intervieivs and 
questionnaires are used in social science research, the process of dM^ 
collection intnjdes itself into the consciousness of the subject and, a^ a 
consequence alters the subject's responses. UnobtruHive rnea^uieB, by 
their nature, avoid mo*^t, if not all of the reactive bias associated with 
Inlerview and i|ue^tjonnaire methodologies. 

Webb et al, (1966) have deiJcribed five categories of unobtrusive 
measures; physical traces (natural erosion or accrcHon processes, ruch an 
the wear on library book pages or the refuse left behind by an earlier 
civilisation), continuous archival records (e g,, actuarial records, govern^ 
ment records); intermittent archives (e,g., wriiten documents, sales 
records): simple obervatlons (eg,, of behaviors), and physical devices 
{e g , cameras, video and audio tapes). 

The measures listed above index some interesting illustrations of 
physical tracer and simple obscrva Jons. For example, the fact that the 
floor tiles around the hatching^chick exhibit require replacement approx- 
imately once every six weeks, compired to a replacement rate of several 
years for the tilep around other exhibits, can be taken as a reasonably clear 
reflection of the relative Interest-value of the exhibit:, So far as the 
shrinking diameter of the circle of children is concerned, if it were also 
known that the shrinkage wa^ observed during a ghost^siory^telling 
sess'^a then the observation would have been recognised for what It isi 
an unobtrusive measure of »He degree of fear induced in the children by 
the stories {and how much more reliable and valid than what the children 
might tell u% ,r asked, "How scared were you?"). As for the dilation of the 
pupils in customers eyes, Chinese jade dealers have usea it as an indicator 
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of ruj^lurtuT inlrrc- iilghicri hvmh haw hvvn 

ohmved io be li- - ;iador mu^t cmtcr the ring. 

ThcTe is no conser f : . rfh : .>wih is tiltrilnitijble Io higher 

anxidy or io : .aher from the mor on Hiosc 

dnys. Probably l^^ pp. v and a). 

Much has ber e methuds of the social sciences 

might be brougl- of uutcomes asHcs^mtTit in higher 

education (e,g., L : . American Collt*gi* le^^ling Program, 

ig$o; Aslin, 197; , - n, however, has been given to Ihc' 

measurement prnK in these methods and to how those 
problems minhf h- . ^ J at least counlerbnlanced. Some critics 

consider the pmrr^i i - ^ a interviews and questionnalre«i to be bo^h 
mww and unn^^rz-iK uy. Vvviib el (h^vI)* for example; 

lament \hh overdependence upon a ?^ingle, fallible meihud. Inter- 
views and que&liunnaire^ (nlrude a foreign element into the social 
selting they would describe, they create a^ well an measure allitudeH, 
they elicit typical roleji and responses, they are limited to those who 
are accessible and will cooperare, and the responses obtained are 
produced in part by dimensions of individual difference irrelevant to 
the topic at hand, 

Bui lite pmcipal ahjeclmt is 0ml Ihey (m mrJ aloHe (p. n emphasis ir 
tlie original), 

Unobtrusive meaBures, such a§ those listed above, offer an Important 
methodological counterweight to the unknown and unbalanced reactive 
bias in intervi^:w- and questionnaire-generated data sets, such as those 
upon which wc now rely to assess the educational outcomes of college. 

The remedy ior these ailments, of course, lies not in the replacement of 
the research tocds now In widespread use. Thin is no call to rally the 
Assessment Luddites. Rather, the intention Is to enconrage outcomes 
researchers to suppkin'.nt standard approaches with methods and mea- 
sures now largely unknown, unconsidered, or Ignored. The purpose, here, 
Is to make "The Case for Unobtrusive Measuresr a id that warrant can be 
argued on at least three grounds (one major, and two secondary): i) 
measurement, 2) cost, and 3) prudence. 

The Measurement Warrant 



The strongest arguments for the use of unobtrusive measures can be made 



(nppropri^iidy enough) on nieti^^rcmunl grounds. Reciitl llnil llio principnl 
abjection of Webb ei .)l { ig66) lu [\w curr<Mil rt'liiinc t» on inliTVttnvH ,uid 
queslionniiims was Hinl ''ihey .vv mvd ^imv ' (p. i), Jhv foundnlion of 
liiis objccHon is ihnh 

Every measurement promiure carries with it certain chnMclerislic 
sources of error,, Jt follows lh*it ihey are in error in different vvays 
and differeiil degrees. The errors we reftT to nre constant wilhiu 
type^j of men^ures— the direction and size of the error nre .issumed 
to be fixed for a given set of measurement operations. I loweven tlie 
direction of errors h assumed to be random fmv55 procedureH. For 
any given measurement tank the errors are additive: an error in one 
direcUnn will tend to fanrel out an error in llie i>ther direclion 
(SeelireNt and Phillip.s. 1970, p. 2). 

Sechrcst and Fhillip!j go on to note problemH occabioneil by differente^ in 
the magnitudes of the errorH involved anil their effects on the precision of 
measufement, but the point is clear and the strongest argument for the one 
of multiple and iiiffvrml mmmtL*^ of the same trait or behavior— what 
Webb et al. (igto) and others {e,g., Campbell & Fiske. 1050) refer to as 
"multiple operalionism;' The intent is to employ multiple measures tliat 
"lihare in the theoretically relevant components (of the trail or behavior 
under study) but have different patterns of relevant components ' Webb 
et al., igbb, p. 3). \Nhm one samples measures, one dlso samples their 
strengths mui their weaknesses. And as in sampling theory, the larger the 
sample size, the greater the reliability of estimation. 

The utility of multiple measures in general and unobtrusive measures 
in particular. Is apparent in another way. Much of the a^search on student 
outcomes, particularly that focusing on institutional contributions to 
student growth, relies on various causal modelling techniques based on 
multiple regression. The multicolinearUy among theoretically indepen- 
dent predictor variables, and the autocorrelations among the same mea- 
sures used over time in longitudinal designs, present welUknoivn, but 
frequently ignored, problems for the interpretation of path coefficients or 
regression weights. The problems of "bouncing betas" and the difficulty 
of replicating most studies in the social sciences are also welNknown. Such 
Interpretive difficulties notwithstanding, however, one researcher (cited 
in Kerlinger k Pedhazur, 1970, p. 446) has suggested that regression 
coefficients give us the laws of science, snd many who employ regression 
analysis, or who read and rely on the results of such studies, may be 
similarly inclined to place more credence in the findings than is warranted. 

SO 



The vvlsdom of iTUilliple--^and uriubtruNivc-^niuasureH h evidenl in 
still other ways, Rcscnrch on the dynamics of atHludc and vnlue fornititlon 
iind change hn both perccplual and bchavioriil dimunsionH. What cone- 
spondcnce e^isl^ between whrtt «1 respondent prafes*jes to believe »ind 
how th*it person nctunlly behaves? Relionce on quest ionnti ires nnd inter* 
views In such investignlions requires an act of faith that the corre8pnn= 
dence is liigh, when the fact of the matter may very well be olherwiHu= 
One can have significantly greater confidence ii the reliabilily and 
validity of interview- or questionnaire^iased claims about attitudeH and 
beliefs if those claims are manifested bchaviorally in natural settingn. Uned 
in this faBhion^ unobtrusive measures constitute a form of convergent 
validation and go a long way toward reducing the irilernal validity 
problems inherent In ex po^t jitcifl research designs. 

Unobtrusive measures have their own limitations, of coursv, lor we 
rarely, if even know itwir characteriBtic sources ot error, rhus. we cannot 
confidently estimate the extent to which use of an iinobtrusive measure 
would be a useful and complementary addition to a series of meaHurement 
procedures or simply increase the error already present. And, like inter- 
view and questionnaire items, to the extenl that unobtnisive measures 
rely on single observations^ thL;^ are likely to be unreliable and, conse- 
quently, of limited validity (Sechrest & Phillips, 1979, pp. 5-7). Despite 
more thafi a two^decade history, much research remains to be done on the 
measurement characteristics of unobtrusive measures. 

Before all hope and confidence in the utility of unobtnj'Jve nieasures is 
abandoned, however, it is useful, at least insofar as the assessment of 
educatignal outcomes is concerned, to differentiate "liaobtrusive mea- 
sures ' as a set of scientific research tools from "unobtrusive measures" as 
a metaphor. In the first instance, it Is quite possible to apply unobtrusive 
techniques and measures in a remarkable variety of experimental studies 
(see Bochner, 1979). As such, the rigor characteristic of true experiments 
can be brought to bear in naturalistic settings (like colleges and universi- 
ties) and threats to internal validity are significantly reduced if not 
eliminated. 

For example, if an institution wished to know the extent to which 
cultural and racial openness was a trait characteristic of the campus, one 
might design a study similar to that reported by Campbell, Kruskal and 
Wallace (1966). In that investigation, the tendency of White and Black 
college students to sit by themselves in racially homogeneous groups in 
classrooms (rather than mixing randomly) was studied as an indicator of 
racial attitudes. 
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While such formal, unoblrusivc rcst'orch cf/orts arc cerlninly possible, 
'hey arc probably not likely to comprise .1 complete or .idequntc out-' 
comes assessment program. "Unobtrusive measures" as a metaphor for 
non-rcactlvc sources of information that already exhl in various forms and 
locations across a campus are more likely to yield useful vehicles of 
assessment. Examples include such standard records as registrar's files, 
disciplinary records. Graduate Record Examination (grb) scores, and 
alumni giving records. The category can also Include less conventional 
measures, however, ranging from transcripts sent to other undorgraduafe 
institutions (student satisfaction), to case loads in the health services and 
psychological counseling service (amount of stress on campus), to library 
usage rates (students' intellectual curiosity), Unabtnisive measures may 
be based on observations as well as records. Such measures In colleges and 
universities might include assessment of a ca.nnus's intellectual climate as 
reyealed on bulletin boards and in grafflli (see ClardI, 1970, for a delightful 
discussion on this topic) and in conversations overheard in a student 
union snack bar. Tlie point to be made Is that unobtrusive measures— 
whether scientifically formal or casual— offer a source of information 
about the educational process and its outcomes that serves a legillmale 
and important measurement role by counterbalancing the systematic 
error characteristic of conventional measurement and research designs 
and by validating information gathered by means of those standard 
procedures. 



The Cost Warranl 

The costs of assessing educational outcomes are little understood. The 
proponents of the "benefits" portion of the cost-beneflts equation have 
been dominant, and only recently has attention been turned to an 
estimation of the other side of the balance scale. How much in the way of 
resources is and should be invested In the production of outcomes 
InformaHon? The question opplles to all Information gathering, of course, 
whether outcomes or otherwise, but costs in other sectors are betted 
understood and estimated than they are in outcomes assessment. The real 
issue, as Ewell and Jones (1986) put it, is! "How much more money 
(beyond that already committed to outcomts-related information gather- 
ing) do we have to spend to put In place an assessment program that Is 
appropriate to our needsr (p, 34). 

Based on a set of assumptions about the nature of the assessment 
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programs likely to be mounted by institutions of varying typan and mm, 
Ewcll and JoneH {igBd) t*stim«itL* Incrcmcmttil costs ranging from $30,000 
(in a small privijte, liberal arts college) to $130,000 (in a major public 
research university), It is important to bear In mind that these are 
incremental, not total cost estimates. It is revealing to notice Ewell and 
Jones' assumption of the use of conventional questionnaires, whether 
commercially available (e*g,, The act's Comf) or locally developed (e.g., 
senior examinations in the major field disciplines)* 

No one has attempted to estimate the incremental costs of assembling 
information unobtrusively, Given the fact that much of this sort of data 
already exists, and given that much of it is electronically stored and 
reirieVtibks it seems reasonable to suggest tnal the costs of unoblrusivt 
measurement and analysis are likely to be lower than those of more 
conventional measures and methods, perhaps significantly lower. There 
is, of course, conMderable room for cost variabilily, but the initial propo- 
sition holds; analyzing data that are already available in one form and 
place or another is likely to be less costly and time-consuming than 
gathering data de novo. 

The Prudence Warrant 

Ewell (1984) has written that "the most vehement objections to the 
systematic a^^essment of institutional impact will come from faculty" 
(p.72). These objections, says Ewell, are likely to derive from either or 
both of two sources: first the fear of being negatively evaluated, and 
second, a philosophical opposition based on the belief that the outcomes 
of college are inherently unmeasurable and that the evidence from such 
studies is "misleading, oversimplifying, or inaccurate" (p. 73). 

To counter faculty opposition, Ewell recommends that persons respon- 
sible for outcomes assessments "recognize publicly the inadequacy of any 
single outcome measure or indicator and , , . collect as many measures of 
program effectiveness as possible" (p. 73). The point is related to the 
argument for unobtrusive measures made earlier on meaiurement 
grounds and h iikely to be recognized and given weight by faculty of all 
disciplines. The effect is likely to be a reduction in faculty resistancp to 
educational assessment. Even if the measurements cannot bf" risily 
explained to non-spclal scientists, most faculty members will : v iliar 
with the concept of "triangulation" in astronomy, as well as in map^ead- 
ing and surveying. The use of multiple measures to portray some educa- 
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iionni oukomc In likely to hnvc a facv validity Ihdi is appcMlinH fttcully 
members. It seems ren^oniible io expecl such ,in effect to influenci* 
poBiUvuly both fncuHy piirUcipiiHon in ouicomes nsse^^smeni progrnms 
and confidence in the conclusionH derived from Ihe evidence assembled, 

Unobtrusive Measures in Higher Education 

Whal are Home unobtrusive measures in higher educition .ind how might 
they enhance our understiinding of various educaUonnl outcomes? Gwll 
(ig&4h following n review of various Btructure.^ and taxonomies, has 
suggested thnt educational aj^ses^menl should focuii on three major areaB; 
knowledge, skllb, and values and nttiludes, wilh a fourth category, 
studcniB' relaUons with various groups in the larger society, reprcsenUnij 
the behavioral manifestations of the first three areas, JuKtapomMon of 
these four dimensions against three of the general types of unobtrusive 
measures described earlier affords a useful framework for thinking about 
the sorts of institutional information that might lie used to aid educational 
aHsesHment. The matrix below is intended to be suggestive, to focug 
thinking on important assessment topics, and, thereby/to highlight the 
potential opportunities to employ unobtrusive measures. 

Types of Unobtruiive Measures 

Otiicotm Cttlc^arm PImkiil Archives Otwrvnfhm 
/r/ia'.s & KvcimU 

Knowledge 
Skills 

Attitudes/Values 
Relations vv/Soclety 



Space predudes discussion of possible measures that might occupy 
e^^^h of the cells in this matrix, and, as will bo seen, the bpundaries 
between the several categories of unobtrusive measures are not always 
precise. Moreover, some of the cells are of greater interest than others, 
and some unobtrusive measures are more readily accessible than others. 
Two cells easily meet both of these criteria, namely, the "Knowledge- 
Archives" cell and the "Attitudes/Valuei^Observations" conjunction, and 
attention will be focused on them for illustrative purposes, beginning 
with the latter of the two cells. 
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T\w ubHcrvatlonal techniques of Cnmpbrll Krusknl nnri Wallace ( i gOO) 
for inferring nKml Miiudm and relnHonn an n c*ifripus hiive been summn' 
rlzed, VariMom an Ihh approach might include n j^ludy of 'Niggregaling'' 
(Campbdl, Kruskan and Wallace, 1966) in dining hnll?; and cafeterias, in 
dmicTB of students studying in the library or gathering in other public 
awm, in institutional residence hall raammate patterns, and in otiier 
institutional settings. 

Something of the importance students attach to the life of the mirul 
might be inferred from several sources, including the number, mu, and 
participation rates in ^irmal student organizations and clubs that have 
some specific academic purpose (e.g,, discipllrte4vised clubs, literary and 
artistic publications, pertormiiig arts groups), as compared with argani^^ii- 
tlons thai: have athletic recreational, entertainment, social or other 
purposes as their princlpale nti^mi tl't'tre. (Sotiie of this in forma I ion might 
be gleaned from records j 

Similarly, inferences about the relative emphasis given to the academic 
and social life of a campus might be made based on an examination of the 
content of campus concert, film, lecture, and speaker series, as well as 
attendance records. For residential campuses, the institution s role in 
students' lives—and Its potential for influence— may be reflected in the 
extent: to which students evacuate the campus for other locations on 
weekends. Clardl (1970) has suggested that the content of graffiti reflect 
the Intellectual tenor of a campus. One might add the content of bulletin 
boards to that reflection. 

The number of students who are registered— and active— voters can 
be taken as a sign of students' interests in, sense of responsibility toward, 
and willingness to participate in the political life of a larger community. 
Or one might explore the level of social responsibility in a student body 
by designing an experiment around the frequency with which students 
returned library books that were presumably 'lost" More simply, the 
proportion of the library s total overdue volumes that are signed out to 
students (or faculty) provides at least one index of the level of simple 
courtesy, if not social responsibility, on a campus. Vandalism, both in 
absolute magnitude and rate of change over time, offers another reflection 
of the quality of life and the attitudes and values prevalent on a campus. 
As suggested earlier, the rates over time at which the health service's 
physicians prescribe stress-related medicines, and variations in the case 
loads of the counseling cQnler staff, might both be used to Index the 
amount of potentially unhealthy^ — and perhaps educationally dysfunc- 
tional— stress in the campus environment. Hodgklnson and Thelin 
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(1970 offer an mprmmn list of olher pnMsibilitii'j^. The varidy is limited 
only by om'ti Um^]\m\\on and ingonuiiy, 

Without question, the major impact of college on students' cugnitlvx* 
development Is delivered through the curriculum, tind any outcomes 
tiBSessment program must deal in one fashion or onolher with the curricu- 
lum and with classroom-based learning. A variety of reactive measures 
have been developed to assess the nature and extent of students' cogni- 
tive growth {e.g. the act^cpmp and Graduale Record EKaminatians 
subject tests), and these measures are typically used in "value-added'^ 
research designs of Varying degrecB of sophistication and validity. 

Warren iigB4) and Pascarella (tgBo) dincus^ some of th e conceptual 
and rnethuuulogical limitations of this approach to educational assess- 
ment, and those critiques need not be reiterated here. The point to be 
nuide h tfiat something of the nature and extent of student learning ran 
also be inferred from unobtrusive measures, frrnn a data base that already 
exists and that has reasonable claims to reliability, namely, the registrar s 
file, which contains extensive information on the courses students have 
taken and the grades received, 

Fincher (104), recognising the weaknesses and disadvantages of the 
grade-point average as a criterion of what has been learned, also marvels 
that "it works as well as it does" (p. 300). He writes: 

, . .the freshman CPA will often display scalar features thar are quite 
remarkable: a tenacious arithmetic mean, a standard deviation of 
about one^-half letter grade, and a range of five or more standard 
deviation units. More remarkable, perhaps, the freshman gpa 
appears to be more Immune to contamination than separate course 
grades are, and it is a relatively independent criteria despite being a 
faulty one. In addition, the freshman gpa is rf^levant to ^uch educa* 
Honal decisions as the dean s list, student probation and dismissal 
the maintenance of athletic eligibility, the continuance of scholar- 
ships, etc. if not a completely adequate criterion of academic per- 
formance, the freshman gfa still serves many educational purposes 
(p. 380I 

Wilson (1983) reporti that admissions measures are essentially as valid 
for predicting long-term cpa as freshman year gpa. Because of this 
property, Fincher suggests, cumulative gpa may yet be a useful measure 
In educational assessment and worthy of analyiis, It might, for example, 
be used as the criterion in regression models and covariance analyses in 
which pre^college academic aptitudes and achievements (and other poten- 
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tiaily confounding Vtirbibles) have been cunltolled in »1 study of tht» 
rc^iidual ViiritifRu in long^xnni gpa aUribuf*ihIc» in HUukmi vf(ot\ mul io 
ins I rue Hon and sludent Ifdrning. Slnilkirly, if pre? I lege precliclors of 
ncfldemic performance arc found to have high multiple correLiHons with 
actual college achievement, reason*iblc suspicions might be r*iiRed about 
the overLip of high school And college coursework (Fincher, 1984). 

The registrar's files offer other possibilities. For example, an eKamini!- 
tion of the distribution of courses taken by %'v£C and type of instruction 
(e.g., lecture, seminar, lab. Independent study) might prove extremely 
revealing of the nature of the foriilal educational process experienced by 
students (e.g,, gf^dwatlng seniors). How many opportunities v^ere there 
for students in small numbers to study with a faculty member? Such a 
review might focus on students' first two years. Do large lecture secfions 
dominate .^tudent^' early Gontacb with faculty and collegiate inr^tructlon? 
What is the relative balance of opportunities for active vs, passive student 
participation in their own learning? While recognldng that "miair is not 
necessarily "belrter/' most faculty and administrators would probably be 
concerned if students' opportunities for small-group instruction were rare, 

Examination of the relative proportional distributions of students 
majoring md gradmlmg in particular disciplines will tell something of the 
nature of the educational program being delivered and comparisons of 
iuch distributions, both one with the other and each over time, will detect 
shifting emphases in what students arc interested in and what the 
institution \% providing. Simjlarly, student retention rates, both within and 
across majors, may yield useful information. While such rates must be 
Interpreted with considerable care, rates occupying one or the other tall 
in the distribution suggest something about students' views of the 
education afforded In those programs. Preciiely what an extremely low 
retention rate means may be open to dispute, but at the very least It calls 
attention to the need for further Investlgatioa 

Transcript analysis affords a more detailed examination of curricula 
structures and student course-taking patterns and brings one still closer to 
the substance of students' formal education. Using this technique, Black- 
bum et al (1976) undertook a national study of changes In degree 
requirementi between 1967 and 1974, exploring the amount, structure, 
and content of general education, and the structure and flexibility In 
e lected major degree programs. They found, for example, that the typical 
baccalaureate degree recipient In 1974, compared to 1967, had taken 
about aa per cent less coursework in general education. 

Calambos et al. {ig%sl in a study of teacher education in the states 
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comprlsjjig the Soutlierr RrgionnI CilucaHon Board, med Ininbcilpt tiniil- 
ym io cumpiifL* llu* coursL*4akirig palleriiH uf leacher t'ducallori md arb 
and sciences degree gradiialcB. They found lhah on llio average, teacher 
education graduates took proportionally fewer general education credits 
in nil areas except the social sciences than did arts and sciences graduates. 
Their analyses aho led them to conclude that "Given latitude, ^ome 
students will ferret out the routes of least resistance to meet their general 
education requirement.^, and then paBs the vvord on to others ' (Galambos 
et nL, igSf. p. 76). Such a finding on an individual campus in likely to be 
justifiable cause for a detailed—and important — review of general 
education courses and requirenient^ 

The State University of New York at Albany used transcript analysis to 
te.st a belief prevalent among faculty and administratorB that students 
were not gaining a ^'general education'' because the only degree r<?quire- 
ments were those of the major program; all other degree credits were 
elective^. The analysis provided information of the average number and 
percentage of course credits taken by graduates of each academic depart- 
ment in each of some 20 content areas. Results indicated that, while 
students in certain major field area^ were appnrently avoiding certain 
content areas (e,g., natural and physical HCiences. or foreign languages), 
the deviation^i from what mojit academicianfi might ron^iider a "general 
education " were by no means so great as had been anticipated. 

A variation on thi§ approach is afforded by the following matrix 
(adapted from Blackburn et al, 1976, who also 01 i t some useful classifica- 
tion rules): 

Per Cent of 
Courses Taken 

Type of Cpimi* Branhli Dcpih 

Required 
Restricted 
Elective 
Full Elective 



Using this matrix, a computer-based analysis of the transcripts ol all 
students, or of sub-groups of students (e.g., ielccted majors, transfers, 
freshmen), would afford several kinds of information. It would reveal 
something of the variety and depth of the course work to which students 
have been exposed during any given period of time in their college 
careers. In addition, it would suggest the relative control over students 
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cour^c-liiking i*Km isi'd by ihv \mii\i\\Un) mil \hv fiKiinr d^'piirirnenl, tuut 
ufw rnjpjii c^pi'i i ( unHidrratjti' VtUiaUuii lU rcJHs t|c^|i,if linen is v\'ch wit t lift 
Ihe mm imiihtUon, Blackburn cl (1970) afiVr 1I mvful variant (if llu' 
above? wMrlK Hinl tlifffrrnhiite!! gcncrjl tHintiillon rcquirunicnb *ind 
coiifHCS from ifiose nf ilic miijor firhl. If *^till tinoHicT Viiriiiiii?!! wm* 
,idopii»d lo lake* inlu »iccnunl ff^/ir/f iht^ i uur.se work wan Itikt'n (i.e , »i 
matrix that has Iht* samu bfuadth and dvpih column?*, but has for it^^ rowH 
the time dimrnsipn of course4iiking. say. lower and upper divisionh 
infcjrmatlon would be gained on wltcihcr sludcnls arc lakini; ' breadib" 
C0ur$i^$ prior to the ^^e lection of a major, or later in their tolle^»e years, 
perhaps /?//rr the major program requirements have already been ?Hiti';fied. 
The timing issue is important to the educationtil purpope^ of "general 
education'' requirements. Do the requirement exi^t to ensuri* thai !*lu- 
dents have a broad eJ^pOBure to the various disrlpKnes and on the ba?*!^ of 
which I hey can make a more infurrncd selection of a major pro^'^ram/ l)r 
are the requlrenients intended primarily to ensure that ^luderds are 
exposed to ti broad intellectual e^periente at ^imfte point brfnre they 
graduate? 

VVtirren {1954) has suggested the analysis might be taketi a step further 
One might be inclined to beiieve, for example, that ?^uch course4aking 
paiiern analysfeB do not provide a suffit lently detailed portrait of *^ludenls' 
actidemic ejcperience, for ^uch analysen tell nothing of what students have 
leamed. Warren suggests that a reasonable approximation of what has 
been learned might be obliiined by reviewing exa mi nation questions and 
major paper assignments in courses that recur in the pattern of require- 
ments for general education or for a specific ma|or field^ — -whether those 
courses are elective or required. As Warren (1984, p, 13) note^: ''No 
pre-enroilment, normativ e, or comparative information need complement 
it. The assertion is simply that Progfam X as typically completed by a 
known number of students produces the described learning^ A certain 
amount of faith is required, of course —faith that examinations and paper 
asstgnments reflect course content and that a passing grade reflecls the 
occurrence of learning above some threshold of acceptability. 

It should be evident by now that researchers in higher education have 
a wide variety of research designs and measures upon which to draw in 
their efforts to assess the outc?jmes of a college education, Thus fan 
however, the record indicates a viihially exclusive reliance on a subset of 
those designs and methods. The purpose of this paper has been to suggest 
ways in which conventional methods of assembling information on 
student growth might be suppiemented in ways that illuminate rather 



than t^hM^lH^ Wvbh vi a\. (iqov>, p, 14) pul il MitTimlly: 

So long m we numimu as mad scienliHl^. m appm^d\ fn turnptUi 
wm that cnnsiilers rompcn^ailng vnor convoi>^in^^ aiHo!u*r,i 
Hon from indiviilu*i!ly runlamm.iiiHl (riUMsura), iIutimn nocriusi^for 
auimn. II h ufily wIihi n*iivi !y folate faillt in ti singli' rnon^urc* 
Hini Ihc miB*»ive priihtt'inH of %imM svivnvv fv*»(Mrcli vilhile Ihu 
Vfilidity of our compariNoris. 
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